This is an OpenGL implementation of the straight-Xlib version included in src/exampleCode/X/Xservers/{xlayerinfo,layerdemo} which accompanied the prog_layers.ps article. Quoting from that dir's README:
Programming X Overlay Windows explains how to use SGI's SERVER_OVERLAY_VISUALS convention to portably create X windows in the overlay planes.
XLayerUtil.c Sample code for routines to portably query overlay visuals described in the prog_layers.ps article. XLayerUtil.h Acommpanying header file. layerdemo.c Sample code demonstrating how to use pure-X overlay windows to achieve transparency effects. xlayerinfo.c Sample code for a program to list a server's visuals including visual layering information.